Papers with masked language modelling task
Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach (2021.emnlp-main)
Copied to clipboard
| Challenge: | cuneiform clay tablets were written in 2500 BCE - 100 CE and are a target of extensive transcription and transliteration efforts due to their deterioration. |
| Approach: | They propose to use a masked language modelling task to complete missing text given cuneiform clay tablets written on cuniform signswedges (2500 BCE - 100 CE) they develop models which automatically complete these missing signs based on contextual cues and greedy decoding schemes. |
| Outcome: | The proposed models perform well on missing token prediction (89% hit@5) despite data scarcity (1M tokens), and human evaluations show that they are able to transcribe texts in extinct languages. |
To Split or Not to Split: Composing Compounds in Contextual Vector Spaces (2023.emnlp-main)
Copied to clipboard
| Challenge: | Contextual word embedding models rely on sub-word tokenization to represent single orthographic words but are often suboptimal in under-resourced contexts. |
| Approach: | They propose to use a masked language modelling task to evaluate the model's performance . they use re-trained tokenizers to pre-split compounds into constituents . |
| Outcome: | The proposed models improve on the masked language modelling task and compositionality prediction by pre-splitting compounds into constituents. |